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Abstract 


Fostering Opportunities is an innovative student engagement program for middle school and high school 
students who have experienced foster care. In 2018, Fostering Opportunities was selected through an 
open, competitive process by the Governor’s Office of State Planning and Budgeting to receive 
implementation funds financed through a hybrid Pay for Success (PFS) approach. This report represents 
the contractual analysis required for the first success payment, as outlined in the PFS contract. 


This study was a randomized control trial. Sixth to 11" grade students in Jefferson County, Colorado, in 
foster care (at entry into the study) were randomly assigned to either the Fostering Opportunities 
intervention (“treatment”) or business as usual (“control”) condition. This analysis estimated the impacts 
of the Fostering Opportunities intervention on (a) school attendance rates, (b) course pass rates, and (c) 
suspensions over the first four semesters of implementation (spring 2019, fall 2019, spring 2020, and fall 
2020). 


The project met the criteria for “Success Level Three” of four with a 27.78% improvement in the average 
number of suspension incidents among those students who were suspended at least once. Attendance 
improvements were modest but did not meet the threshold for “success.” Course pass rates did not 
improve or meet the threshold for “success.” 


In light of evidence that the COVID-19 pandemic substantively influenced the outcomes, 
recommendations focus on the need to (1) fund delivery and evaluation of the Fostering Opportunities 
program for at least an additional year after the conclusion of the PFS project and (2) account for the 
COVID-19 pandemic in the analytic plan for Success Payment Two. Likewise, given the lack of 
improvement in course pass rates, recommendations focus on the need to (1) engage the PFS Operating 
Committee in identifying corrective action strategies, (2) engage specialists in action reviews of students 
with high course failure rates, and (3) update the Fostering Opportunities manual. 
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Introduction 


Fostering Opportunities is an innovative student engagement program for middle school and high school 
students who have experienced foster care. The program is: 


e delivered by one or more education agencies working in close partnership with local child welfare 
agencies. 


e designed to be responsive to changes in participating students’ schools, living situations, 
caregivers, eligibility for services, and child welfare case status. 


e designed to consider the network of people and systems (within and beyond education and child 
welfare) that are important to each student’s attendance, behavior, course completion, and 
engagement in school. 


e designed to provide continuity in supports and services for as long as students need a dedicated 
mentor and advocate to be successful in school. 


Education agencies take the lead on service delivery because eligibility for the program continues beyond 
the closure of child welfare cases. Students with a history of foster care often need trauma-informed 
educational support and mentoring throughout their entire K-12 educational experience. The goal of the 
program is to help youth who have experienced foster care be successful in school and ultimately earn a 
high school credential. 


Pay for Success 


Pay for Success is an innovative contracting model that ties funding for social 
programs to evidence-based programming and positive outcomes. 


The Pay for Success (PFS) model is an innovative approach to financing evidence-based programs that 
shifts risk from traditional funders—typically a government entity—to private investors who provide the 
up-front capital. Key outcomes, or “success measures,” are agreed upon prior to the start of a rigorous 
independent evaluation. Only if the evaluation shows that the program meets these outcomes does the 
government funder repay the initial investment. 


In 2015, the Colorado General Assembly passed House Bill (HB) 15-1317, which authorized the Governor’s 
Office of State Planning and Budgeting (OSPB) to enter into PFS agreements with lead contractors for the 
provision of program-eligible interventions (CRS 24-37-403). HB 18-1323, a Joint Budget Committee bill 
signed into law in April 2018, provides full funding to cover all direct payments and maximum possible 
success payments for these projects, through a series of annual transfers into OSPB’s PFS Contracts Fund. 


Fostering Opportunities was selected in 2018 through an open, competitive process by OSPB to receive 
implementation funds financed through a hybrid PFS approach. In this case, the implementation costs and 
initial risk were shared between the state of Colorado and the Community First Foundation. If Fostering 
Opportunities meets the key outcomes (“success measures”), then OSPB will pay back the investment the 
Community First Foundation made in the implementation of the program, plus a maximum of a 2% return 
on the initial investment. 
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Timeline for Measuring Success 


January 1, 2019 to December 31, 2020 — Key Process Benchmarks: Five key process benchmarks were 
monitored and reported to the Governance Committee on a quarterly basis for the first two years of the 
project. By the end of Year 2, the program met or exceeded all five implementation benchmarks. Program 
implementation fidelity was also assessed in depth in March 2020 and March 2021, demonstrating strong 
adherence to the Fostering Opportunities model both prior to and during the pandemic. 


May 1, 2021 — Success Payment One: This report details the key findings and resulting payments associated 
with Success Payment One. At this point in the project, outcomes are assessed for all study participants 
regardless of how long they have been enrolled in the study. Some study participants have been enrolled for 
one semester; others have been enrolled for up to four semesters. 


The findings should be considered preliminary indicators of success. It was expected at the outset of this 
project that the study would be “underpowered” at this point in time, meaning that there may not be enough 
observations to detect statistical significance. This is among the reasons why all available outcome data are 
used for each student and some levels of success may be achieved without statistical significance. 


October 1, 2022 — Success Payment Two: A future report shall detail the key findings and resulting payments 
associated with Success Payment Two. Results will describe the impact of the Fostering Opportunities 
intervention at one year after enrollment in the study or the treatment group had access to the Fostering 
Opportunities intervention. 


The findings associated with Success Payment Two shall be the ultimate measures of “success” of this PFS pilot 
program. For this reason, the PFS contract allows Community First Foundation to make up any lost principal 
associated with Success Payment One under Success Payment Two repayment triggers. 


Description of the Study 


This report represents the contractual analysis required for the first success payment, as outlined in the 
PFS contract. The purpose was to estimate the impacts of the Fostering Opportunities intervention on 
school attendance rates, course pass rates, and suspensions over the four semesters of implementation 
(spring 2019, fall 2019, spring 2020, and fall 2020). The PFS contract details the triggers for success. 


The study was a randomized controlled trial (RCT) where sixth to 11" grade students who were in foster 
care (at entry into the study) were randomly assigned to either the Fostering Opportunities intervention 
(“treatment”) or business as usual (“control”) condition. The study followed an intent-to-treat model, 
meaning outcomes were evaluated based on the offered service. Therefore, all students who were 
enrolled in the study were included in the analysis, regardless of their level of engagement in the 
intervention. We used outcome data for all four semesters during the study time period to explore how 
the length of time students have access to the program (“term”) interact with the potential effectiveness 
of Fostering Opportunities intervention (“treatment”). 
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Research Question for Success Payment One: 


1. What is the impact of Fostering Opportunities on students’ 
a. attendance rate, 
b. course pass rate, 
c. odds of being suspended, and 


d. number of times being suspended 


at four semesters of implementation’, regardless of length of time since 
randomization? 


The evaluation also included process benchmarks that were assessed throughout the study period and in- 
depth assessment of adherence to the Fostering Opportunities model. 


Process Benchmarks 


e Five process benchmarks were included in the PFS contract and set the 
minimum thresholds for number of youth served, consistent engagement 
of youth, and adherence to evaluation requirements. 


Program Implementation Fidelity 


e Thirteen indicators were used to describe adherence to the Fostering 
Opportunities model and crossed the domains of systems alignment, 
program characteristics, and the role of the specialist. 


The study took place in Jefferson County (Jeffco), Colorado. The intervention was implemented by 
“specialists” hired by the school district who check in weekly with students, ensure caregivers and child 
welfare case workers have timely and accurate information about students’ educational progress, and 
consult with teachers on trauma-informed approaches to help the students be successful in school. These 
specialists follow students through planned and unplanned school changes within Jeffco schools and to 
adjacent school districts. The intervention and the study design assume that some students will transfer 
out of the school district, and procedures are in place to continue some aspects of service delivery and to 
track student outcomes. 


During the study period, the five specialists had an average caseload of 15 students (ranging from 5-23 per 
caseload). The Fostering Opportunities program coordinator provided supervision to these specialists and 
carried a smaller caseload of students. 


'The evaluation plan appended to the PFS contract indicates that the outcomes will be assessed at three semesters 
of implementation. The Governance Committee voted to expand the study timeframe and start counting outcomes 
one semester earlier. This is because there was evidence of high adherence to program implementation fidelity and 
lower than expected enrollment in the study. 
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PFS Success Payment One Key Findings 


Overview of PFS Success Payment Triggers 


Table 1 lists the outcomes that were used as payment triggers for Success Payment One. The numerators 
and denominators for the outcomes consider all semesters for which students were enrolled in the study. 
These outcomes were assessed descriptively and causally (i.e., statistical significance improvement 
associated with treatment effect). 


Table 1: PFS Success Measures 


Success Measure Descriptive Measure for Success Payment One 
1. Attendance Rate The total number of class days actually attended across all students 


in each cohort over all semesters evaluated divided by total possible class 
days across the cohort." 


Suspensions: The total number of students suspended during the semesters evaluated in 
Percentage of each cohort divided by the total students in each cohort. 
students suspended 


SU Tes felicey\-te-}4299) The total number of suspension incidents divided by the total 
Nernbenal number of students who were suspended in each cohort. 
suspensions among 

those students who 

were suspended at 

least once 


Course Passing Rate The number of courses actually passed for all students in each cohort over 
the period covered by the evaluation divided by the total number of 
courses it was possible to pass (i.e., course for which the student received 
a grade) for the full cohort. 


oe (0) ie-le ei te) Cie-lelt-1¢-3| Not measured for Success Payment One per PFS contract. 


Based on outcomes listed above, “success” will be measured for Success Payment One as follows. 
“Improvement” is defined as differences between the treatment and control groups. 


e Success Level One: The project demonstrates either a 5% or greater improvement in any one 
success measure or Statistically significant improvement in any one success measure. 


e Success Level Two: The project demonstrates 5% or greater improvement in any one success 
measure and statistically significant improvement in any one success measure. 


i The PFS contract indicated that attendance rate would be calculated using class periods attended. Those data were 
not available, so days attended was substituted. 
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e Success Level Three: The project demonstrates either of the following: 


o Statistically significant improvement in any two success measures (cannot both be related to 
suspensions) and a 5% or greater improvement in any one success measure. 


o A10% or greater improvement in any one success measure. 


e Success Level Four: The project demonstrates both of the following: 


oO Statistically significant improvement in any two success measures (cannot both be related to 
suspensions). 


o A10% or greater improvement in any one success measure. 


The methods for assessing statistical significance are detailed later in this report. 


Fostering Opportunities Met Criteria for “Success Level Three” of Four 


Success Level Three is defined as the project demonstrating either of the 
__ following: 


e Statistically significant improvement in any two success measures (cannot 
both be related to suspensions) and a 5% or greater improvement in any 
one success measure. 


¢ A 10% or greater improvement in any one success measure. 


Fostering Opportunities met this threshold with a 27.78% improvement in the 
average number of suspension incidents among those students who were 
| suspended at least once. 


The observed percent improvement was not statistically significant. The lack of statistical significance may 
be because the analysis was “underpowered,” meaning that there were not enough students who were 
suspended to be confident that the difference was attributable to the intervention and not to other 
factors or chance (see Methods). Underpowered analyses were expected at this point in the project, and 
observed improvement is considered “success.” 


Attendance Improvements Were Modest and Did Not Meet the Threshold for 
“Success” 


__ There was a 2.76% improvement in attendance rate between the treatment and 
control groups, based on averaging attendance across all semesters. 


The attendance rates for the treatment group were higher than the control group 
| two and three semesters after students gained access to the program. 
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The 2.76% observed improvement in attendance rate between the treatment and control groups includes 
every semester of available data. This means that students who were enrolled in the study early in the 
project (e.g., spring 2019) contribute more to this rate than those that were enrolled late in the study 
(e.g., fall 2020). 


In the statistical model, we found that gains in attendance for the treatment group peaked at two and 
three semesters after providing access to the program. This suggests that it may take more than one 
semester to realize potential benefits of the Fostering Opportunities program on attendance. 


These gains diminished by semester four, but this finding should be interpreted with caution. There were 
relatively few students who had access to the program for four semesters and for all of these students 
whose fourth semester was fall 2020, students experienced a combination of remote and hybrid learning. 
Attendance was similar for the treatment and control groups during this time. 


Course Pass Rates Did Not Improve or Meet the Threshold for “Success” 


There was no improvement in the course pass rate, based on averaging the 
percentage of courses passed across all semesters or tests of statistical 
significance. 


Trends showed a nine percentage point gap between the treatment and control groups for Term 4 (four 
semesters after access to the program), a 14% percent improvement. Term 4 is the only term that did not 
include course passing data from spring 2020, which is when grading practices shifted abruptly due to the 
pandemic and almost all courses were passed by students in the treatment and control groups. 


Even with the context provided by the descriptive trends, the overarching finding warrants further 
discussion to generate hypotheses about how to strengthen program design and child welfare and 
education alignment to improve course pass rates (see Implications section). 


Suspension Incidents Improved and Met the Threshold for “Success” 


Although the percentage of students suspended did not improve, the average 
number of times these students were suspended decreased and met the 
threshold for “success.” 


The percentage of students who were suspended at all was higher in the 
treatment group than the control group by 5.77% across all semesters. Lower 
rates of suspensions are better. 


There was a 27.78% improvement (decrease) in the average number of 


suspension incidents among those students who were suspended at least once 
across all semesters. 
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The observed differences in suspensions incidents in the treatment group relative to the control group are 
based on descriptive analyses. The success in reducing the average number of suspension incidents was 
deemed as practically significant or meaningful in terms of more students on a positive trajectory in school 
prior to the launch of the project. 


While these are preliminary indicators of success, we cannot yet attribute these findings—with 


confidence—to the Fostering Opportunities program because they are not statistically significant. 
Enrolling more students will allow us to establish if causal attributions can be made. 


State-Owed Payments 


Success Level Three was met. The state-owed payment is 75% of the payment 
period principal, which is $427,105. 


The Success Payment was calculated by multiplying the payment period principal ($213,647 + $140,978 + 
$214,848) by 0.75. 


PFS Process Benchmarks 


In addition to the success payment triggers, process benchmarks were set prior to project launch and 
tracked throughout the project period. 


All five process benchmarks were met by the end of Year 2 of the project period. 


The total number of youth served was off track early in the project but met the 
minimum threshold by the end of Year 2. 


The benchmarks related to engagement of youth in the program and evaluation 
requirements were consistently met throughout the project period. 


Table 2: PFS Key Process Benchmarks Results at End of Year 2 


Key Process Benchmark Definition Findings 


1. Total Youth Served Minimum of 30 youth in Year 1 and 60 youth in Year 2 served. MET 


2. Total Youth Served Minimum of 30 youth served for 3+ months in Year 1 and MET 
Continuously minimum of 60 youth served for 3+ months in Year 2. 


3. Total Youth Successful Three check-ins per month for 80% of students continuously MET 
Specialist Check-ins served. 


Us AYU (of of =1-9 101 BY ea Outcome data and necessary intervention implementation MET 
information shared each semester with the evaluator to 

ensure it is collected correctly for the evaluation. Collected 

and shared for 90% of youth served, as certified by the 

evaluator. 


5. Successful Certified as successful by the evaluator. MET 
Randomization 
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Although the minimum number of youth served in the program was ultimately met for the first two years of 
the project, enrollment was not on track to meet this benchmark early in the project period. The PFS 
Governance Committee approved expanding the target population from seventh through 10" graders to 
sixth through 11" graders. Additionally, the Governance Committee approved enrolling students who were 
enrolled in Jeffco Public Schools but in the custody of child welfare agencies outside of Jefferson County. 
Prior to this population expansion students needed to be enrolled in Jeffco Public Schools and in the custody 
of Jeffco Human Services at the time of entry into the study. These corrective actions led to the project 
meeting the minimum threshold for the process benchmark. 


The minimum threshold for enrollment is associated with a PFS clause that could trigger the option for early 
termination of the project. A target for enrollment was also set that was based on a power analysis, or the 
estimated number of study participants needed to detect statistically significant effects of the intervention. 
At the end of the four terms, enrollment fell below its Success Payment One target of 192 students. Through 
fall 2020, a total of 131 students (68% of the target) were enrolled in the treatment and control groups. 


Enrollment was also affected by the COVID-19 pandemic and the drop in the number of youth removed from 
the home by child welfare, an eligibility requirement for participation in the study. Between January 2020 
and December 2020, the total number of referrals received by the Colorado Department of Human 
Services’s (CDHS’s) Division of Child Welfare dropped by 18.5% statewide and by 25.2% in Jefferson County." 
In that same time frame, the rate of children removed from their home dropped from 3.3 removals per 
1,000 children (ages 0-17) in January 2020 to 2.9/1,000 removals in December 2020 across the state. This 
trend was even more pronounced in Jefferson County, with a drop from 2.5 removals per 1,000 children to 
1.9/1,000 removals. 


Although the minimum benchmark for enrollment was met, the sample size through fall 2020 was 131 
students, falling short of the target sample size for Success Payment One of 192 students. Target sample 
sizes were based on a prior power analysis and the number of students that would likely be needed to detect 
statistically significant effects, which are necessary to achieve the highest level of success payments. The 
study was underpowered based on the total number of youth randomized into the spring 2019 through fall 
2020 cohorts. 


The program demonstrated high levels of student engagement for those who did receive services. Through 
fall 2020, 85% of youth participated in the program for at least three months (i.e., continuously served 
process benchmark). These students met with their specialist an average of 3.8 times per month (i.e., 
specialist check-in process benchmark). Data were successfully pulled for over 90% of youth and 
randomization procedures were followed with fidelity. 


The PFS contract requires regular meetings to address operational issues, opportunities, and quarterly 
reporting on key enrollment and fidelity benchmarks to the state of Colorado and the philanthropic funders 
investing in service delivery. This structure, while admittedly cumbersome and time-intensive, has the 
advantage of resourcing and incentivizing addressing barriers to implementation quickly and collaboratively. 


i These data were retrieved from the CDHS Community Performance Center (http://www.cdhsdatamatters.org/) on 
April 19, 2021. 
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Program Implementation Fidelity 


A detailed fidelity checklist was developed for use by Fostering Opportunities leadership and program staff 
to assess their adherence to the model, identify strengths, and engage in continuous improvement. The 
checklist allows for self-assessment along 13 key indicators, listed in Table 3. 


Table 3: Key Program Implementation Fidelity Indicators 


Key Fidelity Indicators Description 


I. SYSTEMS ALIGNMENT 
1. Leadership Framework Evidence of a site’s leadership-driven culture of commitment to 
the Fostering Opportunities program and its principles. 


2. Legal Framework Evidence of a site establishing a strong legal framework for 
Fostering Opportunities so that the program can function as 
smoothly as possible. 


3. Practice Framework Evidence that day-to-day practices and procedures affecting 
students in the school environment are implemented with 
fidelity at this site. 


Il. PROGRAM CHARACTERISTICS 


4. Staffing Evidence that the Fostering Opportunities program is 
adequately staffed at this site. 


5. Database Evidence that a quality Fostering Opportunities database has 
been created and can be used to facilitate network closure at 
the site. 

6. Monthly Progress Evidence that monthly progress monitoring reports are 

Monitoring Report generated every month and shared with both the student and 
all members of the student’s network. 

7. Network Closure Evidence that the program is fulfilling its goal of facilitating 
network closure for participating students. 

8. Equitable Access to the Evidence that students are selected to participate in the 

Program program in an equitable and unbiased manner at this site. 
9. Supervision Specialists are receiving adequate supervision from the program 


coordinator, which enables them to better serve students. 


Ill. THE ROLE OF THE SPECIALIST 


10. Advocacy Evidence that specialists are effective advocates for students at 
this site. 

11. Mentoring Evidence that specialists are effective mentors for students at 
this site. 


12. Social-Emotional Support Evidence that specialists are effective providers of social- 
emotional support at this site. 


13. Academic Support Evidence that specialists are effective providers of academic 
support at this site. 
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Program implementation fidelity was assessed in-depth at two points in time, demonstrating high levels of 
adherence to the Fostering Opportunities model both before and during the COVID-19 pandemic (see 
Appendix A for complete self-assessments). Based on the March 2021 self-assessment completed by 
Jeffco Public Schools, the program met or exceeded expectations along all key indicators, with some 
opportunities for improvement under the Database indicator. A Fostering Opportunities database has 
been created but could be improved to better facilitate network closure through the development of all 
required functionalities, adherence to all relevant federal and local data security requirements, and 
dedicated IT support for technical issues as they arise. 


Discussion and Recommendations 
The COVID-19 Pandemic Substantively Influenced the Outcomes 


Approximately two-thirds of the data used in this report were from spring 2020 and fall 2020. The 
COVID-19 pandemic necessitated remote and hybrid learning during this time. During PFS Operating 
Committee meetings, Jeffco Public Schools shared examples of how the pandemic has affected the 
outcomes of interest, for example: 


e Attendance: During spring 2020, when schools quickly transitioned to remote learning, the way 
attendance was measured varied among schools. Some schools stopped taking attendance. By fall 
2020, there was more consistency in collecting attendance data but transitions in and out of 
remote learning and hybrid delivery continued to affect measurement of this outcome. 


e Course Pass Rate: During spring 2020, the district implemented a policy that grades could only 
improve after the transition to remote learning, but no student’s grade would be lowered after 
that point in time. This likely contributed to extremely high course pass rates for both groups 
during the spring 2020 time period. 


e Suspension Incidents: Throughout the pandemic, there were very few suspension incidents 
districtwide. Students were primarily not physically in school buildings. Zero students in the study 
were suspended during the fall 2020 semester. 


Recommendations 


1. Fund delivery and evaluation of the Fostering Opportunities program for at least an additional 
year after the conclusion of the Pay for Success project. This will allow the potential impact of the 
Fostering Opportunities program to be assessed during time periods that do not include 
transitions in and out of remote and hybrid learning models. 


2. Account for COVID-19 pandemic in the analytic plan for Success Payment Two. 


e Consider adding statistical controls to the models to account for the unique impact the 
COVID-19 pandemic time period may have on outcomes. 


e Consider measuring “percent improvement” in two ways: (1) For the project as a whole 
and (2) For semesters where variation in the outcomes were not influenced by the 
pandemic (e.g., fall 2020 suspension data could be dropped). This could mitigate the 
finding that the percentage of youth in the treatment and control groups affected by the 
COVID-19 pandemic during Term 1 and Term 2 is unbalanced (see Table 4: Enrollment in 


Treatment and Control Groups by Term and Calendar Year). 
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There is No Evidence of Improvement in Course Pass Rates 


The national literature indicates that improving attendance and reducing behavioral incidents are part of 
the foundation for academic success. The results of this study suggest that Fostering Opportunities holds 
promise for improving attendance rates within a year of entry to the program and also reducing the 
suspension incidents. There may be gaps in child welfare and education systems alignment that are 
contributing to static course pass rates. There may be opportunities to strengthen the role of the Fostering 
Opportunities specialist and accelerate progress on course pass rates. 


Recommendations 


1. Engage the PFS Operating Committee in Identifying Corrective Action Strategies. The PFS 
Operating Committee is intended to be a resource to the Fostering Opportunities program 
implementation team to brainstorm how to further align the child welfare and education systems 
to support academic success. 


2. Engage Specialists in Action Reviews of Students with High Course Failure Rates. This process can 
help identify points in time where additional support or intervention may be applied to work with 
future students to prevent course failure. 


3. Update the Fostering Opportunities Manual. More information may be needed in the program 
manual about strategies to reduce risk of course failure. For example, strategies identified through 
recommendations #1 and #2 above could be incorporated into the manual and organized in 
categories such as: resetting after failures in the prior semester, starting semester strong, getting 
back on track mid-semester, and prioritizing at the end of a semester. 
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Methods 


Randomization 


Students in grades 6-11 who were in foster care (at entry into the study) were randomly assigned to the 
Fostering Opportunities intervention (“treatment”) or business as usual (“control”) condition. 
Randomization occurred at the start of each semester using a computer-generated random number. 
Sibling pairs were randomized by alternating the random assignment based on the lowest grade and the 
highest grade of the sibling group. Randomization weights, set on a per cohort (semester) basis, ranged 
from 0.3 to 0.7 probability of assignment to treatment. 


Random assignment procedures were followed with fidelity. There was no 
indication of crossover. 


The randomization procedures were piloted during the building period, fall of 2018. By the time the study 
launched in the spring of 2019, the process and clear paths of communication and timelines with the 
providers and data contributors were established. There has been no indication of crossover since the 
study launched. 
The cut-off dates for being part of a cohort were as follows: 

e Students randomized between August 1 and October 1, were included in the fall cohort. 

e Students randomized by February 15, were included in the spring cohort. 
Treatment (participation in the Fostering Opportunities program) was assumed to have begun 
immediately after randomization. 

e Students randomized after February 15, were included in the next school year’s fall cohort. 


Treatment (participation in the Fostering Opportunities program) was assumed to have begun in August, 
although some initial outreach to families occurred for some students prior to August. 
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Term 


A goal of the Success Payment One analysis was to gain insight into potential variations in outcomes 
associated with the length of time students have access to the Fostering Opportunities program (i.e., time 
since randomization). 


e Term 1: Outcomes for the first semester after randomization. Students enrolled in the treatment 
group had access to the program by October 1 for the fall semester or February 15 for the spring 
semester. 


e Term 2: Outcomes for the second semester after randomization, this would be the first complete 
semester where all students in the treatment group had access to the program prior to the start of 
the semester (e.g., enrolled during Term 1). For some, but not all students, this time period includes 
summer. 


e Term 3: Outcomes for the third semester after randomization. 
e Term 4: Outcomes for the fourth semester after randomization. 


Thus, term is a measure of time since access to the program began (i.e., randomization), not calendar time 
(e.g., fall 2020). 


Enrollment by Term and Calendar Year: COVID-19 Pandemic 


The study was designed to use all available data for the evaluation of Success Payment One. Days 
attended, courses passed, and suspension incidents were counted for each student during every term 
after randomization. The number of students enrolled in the study grew as calendar time passed. This 
means that there were more observations (data points) included in the analysis from more recent 
semesters, which coincided with the COVID-19 pandemic (see Table 4, total observations). 


Approximately two-thirds of the observations during this study occurred during 
the COVID-19 pandemic. 


Thus, the intervention average treatment effects reported in the statistical 
models are heavily influenced by the COVID-19 pandemic. 


Descriptive findings pre-COVID and results for Terms 1 and 2 provide context for 
the potential promise of the Fostering Opportunities program. 


As illustrated in the diagonal of Table 4, the 20 students enrolled in the treatment group and 28 students 
enrolled in the control group during spring 2019 had four observations for each outcome (e.g., an 
attendance rate for Term 1, Term 2, Term 3, and Term 4). Whereas students enrolled later in the study 
had fewer observations. For example, those enrolled in fall 2020 only had data associated with Term 1. 
Practically, this means that approximately two-thirds (~65.5%) of the observations for the study as a whole 
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occurred during semesters affected by the COVID-19 pandemic (i.e., spring 2020 and fall 2020)". Thus, the 
intervention average treatment effects reported in the statistical models are heavily influenced by the 
COVID-19 pandemic. 


Table 4: Enrollment in Treatment and Control Groups by Term and Calendar Year 


Spring 2019 Fall 2019 Spring 2020 Fall2020 Total Term 


Treatment 20 32 55 76 
Term 1 20 12 23 21 76 
Term 2 20 12 23 55 
Term 3 20 12 32 
Term 4 20 20 
Control 28 44 49 55 
Term 1 28 16 5 6 55 
Term 2 28 16 5 49 
Term 3 28 16 44 
Term 4 28 28 
Total Observations 48 76 104 131 


The randomization weight, or odds of each student being assigned to the treatment or control group, 
varied based on availability of Fostering Opportunities specialists to serve youth. This decision to adjust 
randomization weights was guided by the goal of maximizing the number of students offered access to the 
Fostering Opportunities program. Adjusting randomization weights to fill a treatment condition is typical 
in RCTs and was approved by the PFS Governance Committee. 


An unintended consequence of adjusting randomization weights is that the percentage of students in each 
term who were affected by the COVID-19 pandemic is not balanced across the treatment and control 
groups during Terms 1 and 2. 


During Terms 1 and 2, the COVID-19 pandemic may have disproportionally 
affected outcomes for the treatment group. 


e During Term 1, the percentage of students with observations that 
occurred during the COVID-19 pandemic is 37.9 points higher for the 
treatment than the control group. 


e During Term 2, the percentage of students with observations that 
occurred during the COVID-19 pandemic is 20.8 points higher for the 
treatment than the control group. 


’ “Approximately” is used to describe the percent of observations during the COVID-19 pandemic because there are 
differences across semesters and study participants in total number of days possible to attend and courses possible 
to pass associated with where they attended school and their schedule. The two-thirds reported here is the 
cross-walk of enrollment and term reported in Table 4. 
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e During Terms 3 and 4, COVID-19 effects on outcomes are comparable 
between the treatment and control groups but may not be generalizable 
beyond the COVID-19 pandemic. 


e During Terms 3 and Terms 4, all observations occurred during the 
COVID-19 pandemic. 


Table 5: Percentage of Study Participants for Whom the COVID-19 Pandemic Occurred During a Given 
Term, or Semester Since Entering the Study 


COVID-19 COVID-19 COVID-19 


Treatment Control Total Study 
Term 1 57.9% 20.0% 42.0% 
Term 2 63.6% 42.9% 53.8% 
Term 3 100% 100.0% 100.0% 
Term 4 100% 100% 100.0% 


Sample 


Fifty-five students (42%) were randomized into the control group, while 76 (58%) were randomized into 
the treatment group. Table 6 presents key demographic characteristics of the sample. 


Table 6: Key Demographic Characteristics of the Sample 


Sample Size 131 students 


Gender 46.6% female 

53.4% male 
Average age 13.5 years old (with a range of 11-18) 
Primary race/ethnicity 14.7% African American 


41.2% Caucasian 
40.5% Hispanic 
4.6% Another race/ethnicity 


Grade at Enrollment 29.9% in grade 6 
14.9% in grade 7 
9.0% in grade 8 
19.4% in grade 9 
13.4% in grade 10 
13.4% in grade 11 
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Baseline Equivalence of Demographic Measure 


We used administrative data from child welfare to assess baseline equivalence of students between 
treatment and control groups. We compared each group at baseline using discrete demographic measures 
of race and gender. 


Demographics were compared using the Cox index for dichotomous variables. Absolute values of the 
effect size of less than 0.25 were considered equivalent. Testing showed that the groups were equivalent 
on gender, the race/ethnicity categories of Caucasian and Hispanic, and foster care placement in the prior 
year. The race/ethnicity category of African American was just over the equivalency threshold but given 
that this is a preliminary study and the small number of youth in this group (n=18), no statistical 
adjustments were made. 


Table 7: Baseline Equivalence Results 


Control Treatment cox d 
Female 0.49 0.45 0.1- 
African American 0.11 0.16 -0.26 
Caucasian 0.44 0.39 0.10 
Hispanic 0.40 0.41 -0.03 
Other Race/Ethnicity 0.05 0.04 0.09 
Foster Care Placement Prior Year 0.60 0.57 -0.09 


Outcome Measures 


Attendance 


Attendance was defined as yes/no (present or absent) for the day, regardless of reason or 
excused/unexcused status. The attendance measure mirrors Jeffco Public Schools business rules for 
average daily attendance rates. 


Course Passing 


Course passing was defined as a student receiving a letter grade of “D” or higher on a traditional A-F 
grading scale or a number grade of 2 or higher on a 1-4 grading scale. A student was considered as failing a 
course with a number grade of 1 or a letter grade of “F” (failing), “U” (unmet), “NM” (not met), “N” (not 
passing), “ND” (student does not demonstrate attribute), or “I” (incomplete). A course pass rate was 
calculated for each student for each semester post-randomization. These course pass/fail distinctions 
were developed in consultation with Jeffco Public Schools and reviewed by the PFS Operating Committee. 


Suspension Incidents 


The percentage of students suspended at least once was defined categorically as whether or not a student 
had one or more suspensions and is inclusive of in-school and out-of-school suspensions. The decision to 
focus on suspension incidents and not differentiate between type of suspension (in-school vs. out-of- 
school) or number of days is because school-based practices and the use of restorative justice approaches 
can influence the type of suspension and length of time students are suspended. 
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Analytic Approach 


Descriptive Analyses for All Outcomes 
Percent Improvement (Success Measure) 


The percent increase formula was used to determine the descriptive magnitude of improvement for each 
of the outcome measures. Data were used from all four semesters. For attendance and course passing 
outcomes, an increase is considered an “improvement;” whereas for suspension measures, a decrease is 
considered an "improvement.” 


Treatment Value- Control Value 
Percent Increase = ee _,_FKv reer X 100 


Control Value 


Trends Over Time (Informational Purposes) 


All study outcomes were investigated descriptively, examining differences between the intervention and 
control groups over the full study period. Attendance, course pass, and suspension rates were examined 
by group over time, with a focus on “term” or number of semesters since enrollment in the study. The 
goal of reporting these trends is informing program improvement. Information trends by calendar time 
are also presented to offer insight into performance of the Fostering Opportunities program prior to the 
pandemic and transition to remote and/or hybrid school. 


Statistical Significance Testing (Success Measure) 


For all research questions, the threshold for statistical significance was set at alpha = .10,” meaning there 
was a 90% chance that any differences detected were attributable to the Fostering Opportunities 
intervention and not random chance. 


Consistent with the PFS contract, the covariates used in statistical models were finalized prior to the 
researchers accessing outcome data for the students enrolled in the study. Data from the project building 
period were used to determine which covariates should be included. The predetermined threshold for 
inclusion of an individual covariate was set at explaining 10% of the variance in the outcome of interest. 
Thus, the covariates included in each of the models were reviewed by the PFS Governance Committee 
prior to study outcome data being shared with the evaluators. 


’ As indicated in the Pay for Success contract. 
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Attendance 


An attendance rate was calculated for each student for each semester post-randomization. “' We used a 
mixed beta regression model”, including a normal random effect to account for the repeated observation 
of the same students and a normal random effect to account for similarities among known siblings in the 
study. The primary regressor of interest was an indicator of participation in the intervention (“group”). 
Other covariates included the time since the semester the student was randomized into the study 
(“term”), grade level at randomization, and an interaction between group and term. 


Course Passing 


We used a mixed beta regression model," including a normal random effect to account for the repeated 
observation of the same students, and a normal random effect to account for similarities among known 
siblings in the study. The primary regressor of interest was an indicator of participation in the intervention 
(“group”). Other covariates included the time since the semester the student was randomized into the 
study (“term”), grade level at randomization, and an interaction between group and term. 


Suspension Incidents 


We used a mixed hurdle Poisson regression model,” including normal random effect to account for the 
repeated observation of the same students, and a normal random effect to account for similarities among 
known siblings in the study. The primary regressor of interest was an indicator of participation in the 
intervention (“group”). Other covariates included the time since the semester the student was 
randomized into the study (“term”), grade level at randomization, an interaction between group and term, 
and foster care placement in the year prior to entering the study. 


“ The definition for attendance, course pass, and suspension rates mirror the PFS contract, page C-1. 

vill As attendance was initially expected to be recorded as yes/no, logistic regression was proposed in the original 
evaluation plan. As a ratio of classes attended, logistic regression was not appropriate. Likewise, the extreme 
skewness in the attendance data would have made linear regression misleading. Therefore, to produce appropriate 
predictions and assessment of effects, beta regression was selected as the most appropriate model. 

vii Similar to attendance, course passing was initially expected to be recorded as yes/no and logistic regression was 
proposed in the original evaluation plan. As a ratio of courses passed, logistic regression was not appropriate. 
Likewise, the extreme skewness in the attendance data would have made linear regression misleading. Therefore, 
to produce appropriate predictions and assessment of effects, beta regression was selected as the most appropriate 
model. 

As noted in the original evaluation plan, a mixed hurdle Poisson regression model is best suited for a count 
outcome with a high number of zero values, as was the case with suspensions. 
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Results 


For each outcome area descriptive information associated with success measures is presented first. Then, 
trends that are reported for informational purposes only are described. Finally, the results of the statistical 
models are reported. 


Research Questions 


Research Question 1A: Attendance 
Descriptive Analysis of Percent Improvement (Success Measure) 


The percent improvement in attendance rate did not meet the PFS contract threshold for triggering a 
success payment. 


There was a 2.76% observed improvement in attendance rate. 


The attendance pass rate for the treatment group was 82.93%. 11,933.6 days were attended by students 
in the treatment group out of a total of 14,390 possible days. The attendance pass rate for the control 
group was 80.70%. 9,608.1 days were attended by students in the treatment group out of a total of 11,906 
possible days. The pass rate for the treatment group was 2.76% better than the control group. The 
percent improvement in attendance rate did not meet the 5% or greater threshold for triggering a success 
payment. 


Trends Over Time (Informational Purposes) 


There was a 6.25% observed improvement in attendance rate at Term 2. 


The gap between the treatment and control groups for attendance varied across terms. At Term 1, the 
attendance rate for the control group was higher than the rate for the treatment group. At Term 2, the 
attendance rate for the treatment group surpassed that of the control group. The gap between the 
treatment group and control group was greatest at Term 2. This finding suggests that two semesters of 
access to the Fostering Opportunities program may be needed to drive practically significant 
improvements in attendance. 
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Figure 1: Trends in Attendance Rate by Group and Term 
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Note. The sample sizes (n) reported here reflect the number of students with attendance data for a given term. 


The calendar year trends illustrate a decline in attendance rates for both the treatment and control groups 
during the semesters affected by the COVID-19 pandemic. 


Figure 2: Trends in Attendance Rate by Group and Calendar Year 
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Note. The sample sizes (n) reported here reflect the number of students with attendance data for a given term. 
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Statistical Model of Attendance Rate 


Although Fostering Opportunities demonstrated promise for improving attendance by a year after 
access to the program, it did not meet the PFS contract threshold of a statistically significant impact 
regardless of length of time since randomization. 


Attendance dropped for the control group at Terms 3 and 4, suggesting a possible 
“pandemic effect.” 


During Terms 2 and 3, the attendance rates were higher in the treatment group 
than the control group. 


For the mixed beta regression model with attendance rate as the longitudinal outcome, results are shown 
in Table 8. Because of the use of indicators and interaction terms for both Group and Term, the Group 
effect represents differences between control and treatment during the first term; the Term effect is 
interpreted as changes over time for the control group; and the Group by Term interaction represents the 
difference in changes over time between treatment and control. 


Table 8: Results of Attendance Rate Mixed Beta Regression Model 


Variable Effect | Standard Error p-Value 


Treatment -0.240 0.231 0.297 
Control (Ref) -- -- -- 
Term 1 (Ref) -- -- 
Term 2 -0.267 0.197 0.175 


Term 3 -0.359 0.202 0.076 
Term 4 -0.689 0.258 0.008 


Treatment x Term 1 (Ref) = Ee == 


Treatment x Term 2 0.502 0.263 0.056 
Treatment x Term 3 0.684 0.304 0.024 
Treatment x Term 4 0.287 0.361 0.426 


Grade 6 (Ref) -- -- 
Grade 7 0.119 0.255 0.639 


Grade 8 -0.080 0.319 0.802 
Grade 9 0.148 0.221 0.502 
Grade 10 0.033 0.266 0.903 
Grade 11 -0.149 0.259 0.565 
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Attendance dropped for the control group at Terms 3 and 4, suggesting a possible “pandemic effect” 


Term is the difference over time for the control group. Using a statistical significance level of alpha = 0.10, 
both Term 3 (B = -0.359, p = 0.076) and Term 4 (B = -0.689, p = 0.008) show statistical significance, each 
with a negative effect. This generally means that attendance rate is expected to be lower for Term 3 and 
Term 4, as compared to the Term 1, for the control group. Specifically, the odds of attending is expected 
to decrease by a multiple of exp(-0.359) = 0.698, or to 69.8% of the previous odds of attending for Term 3 
as compared to Term 1. Similarly, the odds of attending for Term 4 as compared to Term 1 is expected to 
decrease by a multiple of exp(-0.689) = 0.502, or to 50.2% that of Term 1 


Practically, these term effects are difficult to interpret in a meaningful way because all of the observations 
associated with Term 3 and Term 4 occurred during the COVID-19 pandemic, while less than half (42%) of 
the observations associated with Term 1 occurred during the pandemic. The significance of Terms 3 and 4 
could simply suggest a possible “pandemic effect” on attendance. 


During Terms 2 and 3 the attendance rates were higher in the treatment group than the control group. 


In general, the model showed a significant decrease in attendance rate for the control group from Term 1 
to both Terms 3 and 4, while for the treatment group, there was a significant increase in attendance from 
Term 1 to Terms 2 and 3. These effects are consistent with what is seen descriptively in Figure 1, which 
shows mean attendance rates among students in each group for each term of participation. 


The more technical explanation is that both Term 2 by Treatment (B = 0.502, p = 0.056) and Term 3 by 
Treatment (B = 0.684, p = 0.024) interactions show statistical significance, with positive effects. Very 
broadly, these positive interaction effects suggest that the decrease in attendance rate seen in the control 
group is expected to be reduced for the treatment group. For the treatment group, the attendance rate at 
Term 3 is expected to increase by a multiple of exp(-0.359 + 0.684) = 1.384, or by 38.4% of the original 
odds of attending, compared to the first term. Note that this is in comparison to a decrease by 0.698 for 
the control, a difference that is significant. Similarly, the odds of attending at Term 2 is expected to 
increase by a multiple of exp(-0.267 + 0.502) = 1.265, or by 26.5% of the original value, compared to 

Term 1. 


Research Question 1B: Course Passing 
Descriptive Analysis of Percent Improvement 


The percent improvement in course pass rate did not meet the PFS contract threshold for triggering a 
success payment. 


There was no observed improvement in the course pass rate. 
The course pass rate for the treatment group was 81.45%. 1,364 classes were taken by the students in the 
treatment group and 1,111 were reported as passed. The course pass rate for the control group was 


81.80%. 1,352 classes were taken by the students in the treatment group and 1,106 were reported as 
passed. Practically, there was no observed difference (0.35 percentage points or 0.4 percent difference). 
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Trends Over Time (Informational Purposes) 
Grading practices applied during spring 2020 may have affected outcomes by 
term differently. 


The widest gap between treatment and control groups for course pass rate 
_) occurred in Term 4, the only term without data from spring 2020. 


The gap between the treatment and control group for course pass rate was widest at Term 4. This is the 
group of students who were enrolled in the study during spring 2019 and their Term 4 took place during 
the fall of 2020, as illustrated in Figure 3. 


Figure 3: Trends in Course Pass Rate by Group and Term 
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Note. The sample sizes (n) reported here reflect the number of students with course pass data for a given term. 


Terms 1, 2, and 3 contained observations from the spring of 2020 (see Table 4). The course pass rate was 
very high for both groups (near 100%) in spring 2020 likely due in part to grading practices that were 
applied as a result of the pandemic. The course pass rate dropped sharply for both groups in fall 2020 
(at/below 70%). 
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Figure 4: Trends in Course Pass Rate by Group and Calendar Year 
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Note. The sample sizes (n) reported here reflect the number of students with course pass data for a given term. 


Statistical Model of Course Pass Rate 


The Fostering Opportunities did not meet the PFS contract threshold of a statistically significant impact 
on course pass rate, regardless of length of time since randomization. 


) There was no statistically significant difference in course pass rates attributable 
to the Fostering Opportunities intervention at any point in time in the study. 


During Term 4 the course pass rate dropped significantly for all students - 
treatment and control. All of these observations occurred during fall 2020, when 
the semester began with remote delivery of courses. 


There was no difference in course pass rates attributable to the Fostering Opportunities intervention at 
any point in time in the study. 


For the mixed beta regression model with course pass rate as the longitudinal outcome, results are shown 
in Table 9. Because of the use of indicators and interaction terms for both Group and Term, the Group 
effect represents differences between control and treatment during the first term; the Term effect is 
interpreted as changes over time for the control group; and the Group by Term interaction represents the 
difference in changes over time between treatment and control. Group and Group by Term both yielded 
effects that were not statistically significant. 
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Table 9: Results of Course Pass Rate Mixed Beta Regression Model 


Variable Effect | Standard Error p-Value 
Treatment -0.108 0.226 0.633 
Control (Ref) -- -- -- 

Term 
Term 1 (Ref) -- -- -- 
Term 2 0.001 0.252 0.997 
Term 3 0.116 0.262 0.657 
Term 4 -0.593 0.343 0.084 


Group x Term 
Treatment x Term 1 (Ref) -- ne -- 


Treatment x Term 2 0.214 0.336 0.525 
Treatment x Term 3 0.108 0.374 0.773 
Treatment x Term 4 0.515 0.485 0.288 


Grade 6 (Ref) -- -- -- 


Grade 7 -0.259 0.210 0.217 
Grade 8 -0.262 0.285 0.359 
Grade 9 -0.087 0.193 0.652 
Grade 10 -0.089 0.224 0.692 
Grade 11 -0.338 0.244 0.165 


During Term 4, the course pass rate dropped significantly for all students—treatment and control. All of 
these observations occurred during fall 2020, when the semester began with remote delivery of 
courses. 


Using alpha = 0.10 as the statistical significance threshold, the only effect that shows significance is for 
Term 4 (B =-0.593, p = 0.084). For the control group, compared to Term 1, the odds of passing a course in 
Term 4 is expected to decrease by a multiple of exp(-0.593) = 0.553, or to 55.3% the original odds of 
course passing. The non-significance of the corresponding interaction term suggests this effect does not 
differ for the treatment group. This is consistent with the effect seen in Figure 3, as the Term 4 course pass 
rate appears to drop for both groups. 
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Research Questions 1C and 1D: Suspension Incidents 
Descriptive Analysis of Percent Improvement (Success Measure) 


The percent improvement in suspensions met the PFS contract threshold for triggering level three 
success payment. 


Descriptive improvement in suspension incidents was measured in two ways: 


1. The percentage of students who were suspended at all was higher in the 
treatment group than the control group by 5.77 percent. Lower rates of 
suspensions are better, so this is not an improvement. 


2. There was a 27.78% improvement (decrease) in the average number of 
suspension incidents among those students who were suspended at least 
once. 


These findings reflect the observed differences between the treatment and 
control groups and are not causal conclusions about the effectiveness of the 
| program. 


Percentage of Students Suspended 


The percentage of students who were suspended in the treatment group was 5.77 percent higher than the 
control group. 23.63% of the control group was suspended at least once (13 of 55 students). 25.00% of the 
treatment group was suspended at least once (19 of 76 students). 


Average Number of Suspension Incidents Among Those Students Suspended At Least Once 


There was a 27.78% improvement (decrease) in the average number of suspension incidents. 2.77 is the 
average number of suspension incidents in the control group among those students who were suspended 
at least once. There were 36 incidents among 13 students in the control group. 2.00 is the average 
number of suspension incidents in the treatment group among those students who were suspended at 
least once. There were 38 incidents among 19 students in the treatment group. 


Trends Over Time (Informational Purposes) 


_ | While there was a decrease (improvement) in percentage of students suspended 
| was observed during Terms 1 and 2, which are the only terms with pre-COVID-19 
pandemic data. 


The higher rate of suspensions among the treatment group during Term 3 
occurred during the spring of 2019. Term 3 accounts for the overall higher 
percentage of students suspended in the treatment group reported as part of the 
success measures. 


There were very few suspensions in the treatment or the control groups during 
Term 4 and calendar semester fall 2020. 
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Figure 5: Trends in Rates of Suspension Incidents by Group and Term 


SUSPENSION TRENDS IN INCIDENT RATES (TERM) 
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Note. The sample sizes (n) reported here reflect the number of students each term for who suspension data were 
available. 


Figure 6: Trends in Rates of Suspension Incidents by Group and Calendar Time 
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Note. The sample sizes (n) reported here reflect the number of students each term for who suspension data were 
available. 
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Statistical Model of Suspensions 


The Fostering Opportunities intervention did not meet the PFS contract threshold of a statistically 
significant decrease in suspensions, regardless of length of time since randomization. 


| | There was no difference in the likelihood or frequency of suspensions 
| attributable to the Fostering Opportunities intervention, regardless of length of 
| time since randomization. 


For the mixed hurdle Poisson regression model with suspensions as the longitudinal outcome, results are 
shown in Table 10. Unlike attendance and course pass rate models, Term was treated as a continuous 
variable due to convergence issues with the categorical version of term. Term 4 for both groups included 
no variation due to the drop of suspensions to zero; therefore, a statistical between-group comparison 
could not be made. Group and Group by Term both yielded effects that were not statistically significant. 
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Table 10: Results of Suspensions Mixed Hurdle Poisson Regression Model 


Term 


Foster Care Prior Year 


Variable Effect | Standard Error p-Value 
Model for Likelihood of Suspension 
Group 
Treatment -0.696 0.763 0.362 
Control (Ref) -- -- -- 


Term (Continuous) -0.789 0.319 0.013 
Group x Term 
Treatment x Term 0.474 0.405 0.242 


Treatment 
Control (Ref) 


-0.666 


Yes 0.382 0.363 0.293 
No (Ref) -- -- -- 
Grade 6 (Ref) -- -- -- 
Grade 7 0.194 0.491 0.693 
Grade 8 0.379 0.588 0.519 
Grade 9 0.269 0.446 0.545 
Grade 10 -1.947 1.059 0.066 
Grade 11 -0.549 0.686 0.424 


Model for Frequency of Suspensions 


1.068 


0.533 


Term 


Term (Continuous) 


Group x Term 
Treatment x Term 


Foster Care Prior Year 


Yes -0.475 0.502 0.343 
No (Ref) -- - 7 
Grade 
Grade 6 (Ref) -- -- -- 
Grade 7 1.294 0.685 0.059 
Grade 8 1.588 0.753 0.035 
Grade 9 0.708 0.739 0.338 
Grade 10 1.335 1.205 0.268 
Grade 11 -17.522 >100 0.998 


The hurdle model includes two sub-models that address two different aspects of the outcome. The first 
addresses the likelihood of being suspended at all, while the second considers the frequency of 
suspensions for those who are suspended. First, the likelihood of any suspensions shows significance for 
Term, with a negative effect (B = -0.789, p = 0.013). This generally means that the chance of suspension is 
expected to decrease as terms pass. Specifically, the odds of being suspended at all is expected to 
decrease by a multiple of exp(-0.789) = 0.454, or a reduction to 45.4% of the original odds, with each 
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passing term. This is consistent with the plot of mean numbers of suspensions over time shown in Figure 
6, as suspensions drop to zero for all students. Other than grade level, no variables showed significance. 
Tenth graders were more likely than sixth graders to be suspended at least once. 

As shown in the second part of the model, the frequency of suspensions for those students who were 
suspended does not differ by group (treatment versus control) or by Term. Among middle school students 
who were suspended at least once, seventh and eighth graders were more likely to have a higher 
frequency of suspensions than sixth graders. 


Limitations 


COVID-19. The pandemic abruptly changed how schools deliver an educational experience to students. 
The percentage of observations that occurred during the pandemic was higher for the treatment group 
than the control group at Term 1 and Term 2. Thus, it is likely that the pandemic had a differential effect 
on the outcomes for these groups. The Term 3 and Term 4 outcomes only represent semesters that 
coincided with remote and/or hybrid learning during the COVID-19 pandemic, and thus, may not be 
reflective of program outcomes pre- or post-COVID. 


Low Study Enrollment. The study is underpowered, meaning that there were not enough students 
enrolled in the study to allow us to determine if differences between the treatment and control groups 
could be attributed to the Fostering Opportunities intervention with confidence. 


Missing Data. Some outcome values were missing from the final data collected. About 6.1% of values for 
each outcome (attendance rate, course pass rate, and number of suspensions) were missing, meaning 
data were not available for around 6.1% of the total number of times of observation across all subjects. 
Given this very low rate of missingness, models were not adjusted for missing data. Exploring the missing 
values shows that a plurality of missing outcomes were for 11" grade students (30 out of 66), anda 
majority were for students from the first cohort of data collection (55 out of 66). 


Conclusion 


Fostering Opportunities is an innovative student engagement program for middle school and high school 
students who have experienced foster care. The preliminary evidence from this RCT suggests that the 
program has promise for improving attendance and behavior, but not course pass rates, among youth 
who experience foster care. 


In light of evidence that the COVID-19 pandemic substantively influenced the outcomes, 
recommendations focus on the need to (1) fund delivery and evaluation of the Fostering Opportunities 
program for at least an additional year after the conclusion of the PFS project and (2) account for the 
COVID-19 pandemic in the analytic plan for Success Payment Two. Likewise, given the lack of 
improvement in course pass rates, recommendations focus on the need to identify opportunities to 
further align child welfare and education practice and the role of the specialist to support academic 
SUCCESS. 
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Appendix A: Self Assessments 


Fostering Opportunities Self-Assessment Checklist 


This is a self-assessment checklist that can be used by Fostering Opportunities leadership and staff and 
involved child welfare and education leadership to identify areas of strength and areas for improvement 
in the implementation of the Fostering Opportunities program. 


Rater: 2019/20 Fostering Opportunities Team — 1 Coordinator (Sections 1-5), 4 Specialists (Sections 6-13) 
Date: March 2020 
Program Location/Site: Jeffco Schools, Student Engagement Office 


1. SYSTEMS ALIGNMENT: Systems alignment refers to the education and child welfare policy and 
practice coordination that is the foundation for successful implementation of the Fostering 
Opportunities intervention. 


1. Leadership Framework — Evidence of a site’s leadership-driven culture of commitment to the Fostering 
Opportunities program and its principles. 


X Leaders within the child welfare and education agencies position the program for sustainability (e.g., 
support efforts to secure funds, partner in removing barriers, and prioritize internal resources as needed). 


X Leaders within the child welfare and education agencies engage with each other to create solutions when 
implementation challenges arise (e.g., articulate concrete examples). 


X Leaders participate in regularly scheduled meetings to launch and sustain the program. 


Leaders spearheading the implementation have a high level of decision-making authority to prioritize 
practice changes necessary to align systems (e.g., no approval for decisions is needed). 


X Leaders ensure child welfare caseworkers and school leaders (e.g., principals) are familiar with the Fostering 
Opportunities program and how to make referrals. 


Comments: 

- The program is directly supported in the School District by the Director of the Student Engagement 
Office who is a direct report to the Chief of Student Success. Sustainability planning has been in 
progress since the advent of this program. Likewise, the program is supported by a Program Supervisor 
in DHS who is a direct report to the Executive Leadership team at Jeffco DHS who are aware of and 
invested in this programming. There is consistent participation in operations and governance meetings 
to create solutions and plan for sustainability by leadership in both organizations. 


- We purposefully did not check the ‘high level of decision making’ box as the disclaimer that ‘no 
approval for decisions is needed’ does not realistically represent well the systems level changes that 
this program has initiated. Based on the School District governance structure at times the 
improvements initiated by this program have required Cabinet, Superintendent and Board of 
Education approvals. 
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- Leaders have ensured through their own practice as well as in the work of their teams that child 
welfare caseworkers and school leaders are familiar with this program. One caveat is that part of this 
work has been to inform stakeholders that they cannot make referrals due to the nature of the study 
design. 


2. Legal Framework — Evidence of a site establishing a strong legal framework for Fostering Opportunities 
so that the program can function as smoothly as possible. 


X_ The site has established memorandums of understanding (MOUs) that clarify the responsibilities of the 
child welfare and education agencies within the Fostering Opportunities program, including related to the Best 
Interest Determination (BID) process. 


X The site has established data sharing agreements (DSAs) that detail what information can be shared 
between agencies for the purposes of delivering and evaluating the program. 


X The MOUs and DSAs are defined by the target population and geographical boundaries for the intervention 
(e.g., youth in foster care age 11 to 16 who are enrolled in a given school district). 


X The site has developed releases of information (ROI), which are student-specific documents that indicate 
what information can be shared, with whom, and for how long. ROls are on file for all participating students. 


Comments: 

- This program has an established contract that defines the target population and boundaries. The 
program has MOUs as well as DSAs to allow for operations related to information sharing between 
agencies, BIDs, and student, caregiver access to information. There is work that still needs to be done 
to recognize the needs in these areas now that programming is operational as opposed to in its 
conceptual infancy. 


- This program has developed unique Release of Information Sharing protocols attentive to ensuring 
youth are empowered to make information sharing decisions and incorporates the appropriate agency 
specific releases to eliminate any barriers. 


3. Practice Framework — Evidence that day-to-day practices and procedures affecting students in the school 
environment are implemented with fidelity at this site. 


X_ The site has in place standard and, ideally, automated processes for notifying the Fostering Opportunities 
staff when a placement change occurs. 


X Fostering Opportunities’ role in the BID process is implemented as it is defined in the MOU, which may 
include: providing timely and accurate information on students’ educational progress and needs; planning for 
transitions when a school change is necessary; and ensuring communication between human services agencies 
and schools so that seamless transition occurs. 


Comments: 
- Coordinator receives all Notification of Placements from Jefferson County Human Services. 


- The Coordinator will share any notifications with appropriate Specialists when a program student is 
involved. 
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- Specialists play a crucial role in the BID process, including (but not limited to) information sharing, 
transition planning and transportation logistics. 


Il. PROGRAM CHARACTERISTICS: The site-specific characteristics, procedures, and plan of action in place 
to implement the Fostering Opportunities program. 


4. Staffing — Evidence that the Fostering Opportunities program is adequately staffed at this site. 


X There are a sufficient number of staff dedicated to Fostering Opportunities implementation at this site, 
including: child welfare and education leadership, program coordinators, and specialists. This is measured in 
terms of the number of FTEs needed to support the program’s complexity and size at this site. 
o The recommended ratio of students to specialists is 20 students or fewer per specialist. The ideal ratio 
of specialists to program coordinators is 10 specialists or fewer per program coordinator. 


X_ There is a strong plan in place to handle planned and unplanned specialist staffing transitions. 


Comments: 
- This site employs 1 Program Coordinator and 4 Specialists. 


- Coordinator is supervised by Student Engagement Office leadership, including the Director and 
Manager. 


- An Operations committee provides monthly oversight and includes leadership from local and state 
human service agencies, the school district, program evaluators and funding partners. 


- A Governance Committee meets quarterly review program benchmarks, financials and future 
planning. 


- Specialists have the capacity to caseload up to 20 students. Today, caseloads are approximately 50% 
full. 


5. Database — Evidence that a quality Fostering Opportunities database has been created and can be used 
to facilitate network closure at the site. 


X  Adatabase specific to Fostering Opportunities has been created for the site with the required 
functionalities: 


Directory Functionality 


X Educational Progress Monitoring Functionality 
X Child Welfare Information Functionality 


Permissions Functionality 
Reporting Functionality 


Information in the database is up-to-date and accurate. 


The database adheres to all relevant federal and local data security requirements. 
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X__ There is reasonable IT support provided for any technical issues that may arise. 


Comments: 
- This site uses a CAT overlay designed specifically for Fostering Opportunities. 


- Specialists are able to access some of the functions listed above. Not all of the functions are available all of the 
time. 


- The Program Coordinator assumes, but cannot say for sure, that it adheres to all relevant federal and local 
data security requirements. 


- The Program Coordinator has access to reasonable IT support for any technical issues that may arise. The 
Program Coordinator might assume the modernization of TRAILS has taken most of the attention of the IT 
support team. 


6. Monthly Progress Monitoring Reports — Evidence that monthly progress monitoring reports are 
generated every month and shared with both the student and all members of the student’s network. 


X A progress monitoring report is generated by the database for all students every month. 

X Progress monitoring reports are shared and discussed with all students during at least one check-in per 
month. 

X Progress monitoring reports with students’ entire networks as listed on the social capital assessment (and 
the program coordinator may be copied on emails to the student’s network to ensure this is completed). 

X Progress monitoring reports use primarily strengths-based and trauma-informed language. 


Comments: 
-Specialists encourage students to engage and participate in the progress monitoring conversation 
each month. This can occur during one meeting or across multiple meetings. 


-Specialists complete progress monitoring in the database for each student every month. 


-Specialists download the reports from the database and send to each member of the student’s 
network that has privilege to the information. 


-Students receive a copy of the report upon request. 


-Specialists use trauma-informed, person-first, strengths-based language in their conversations with 
students and in completing the reports. 


-Program Coordinator is able to view all progress monitoring reports through the database to ensure 
monthly completion. 


7. Network Closure — Evidence that the program is fulfilling its goal of facilitating network closure for 
participating students. 


X There are consistent documented improvements in students’ social capital assessments over time (e.g., at 
the six-month point as compared to the one-month point there are more people listed in the “supports” column 
or the quality of the support has improved). 
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X Monthly progress monitoring reports are sent to all members of students’ networks (and the program 
coordinator may be copied on emails to the student’s network to ensure this is completed). 


X Specialists demonstrate familiarity with all members of students’ network (e.g. names, details about 
relationship, etc.). 


X Specialists are consistently checking-in with multiple members of students’ networks at least once a month. 


Comments: 

- Specialists are familiar with and communicate with appropriate members of students’ network on an 
ongoing and consistent basis throughout the month. The Specialist employs a variety of ways to 
communicate to ensure information is accessible to everyone. 


- As students identify or are assigned additional members to their network, the Specialist will take 
necessary steps to get a signed ROI, obtain contact information, outreach to and include them in future 
communications. 


- Specialists always share completed progress monitoring reports at the end of each month with 
appropriate members of students’ network. 


- Completing a formal social capital assessment is not a current practice. Improvements in students’ 
social capital are recognized by the Specialist during meetings with the students and documented in 
case notes and progress monitoring reports. 


8. Equitable Access to the Program — Evidence that students are selected to participate in the program in 
an equitable and unbiased manner at this site. 


There is a start list from which to select students to the program that is comprised of those students in the 
school district who are also identified as having experienced foster care. If there are insufficient resources 
to serve all students in the district, participants are randomly selected into the program. 


OR 


Multiple methods are used to identify and recruit students so that the students who are selected to the 
program are not just those who have the strongest advocates in their networks. 


Comments: 
- Unsure if either of the above options accurately describe access to the program. 


- Students are selected to participate in an equitable and unbiased method that supports the 
evaluative nature of the current grant. 
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- Students in grades 6th through 11th with open Dependency and Neglect cases with a local Human 
Services agency are filtered into the database. The Program Coordinator confirms enrollment in Jeffco 
Public Schools and then submits de-identified student information to the Colorado Evaluation & Action 
Lab via a secure link. 


- Students are assigned to the intervention or control group through a formal randomization process 
by the Colorado Evaluation & Action Lab. 

- Students identified for the intervention group are assigned to a Specialists caseload and services begin 
upon receipt of the appropriate county’s consent form. 


9. Supervision* — Specialists are receiving adequate supervision from the program coordinator, which 
enables them to better serve students. 


Note: The program coordinator can give a copy of this section to each specialist to fill out anonymously 
as a way to seek feedback on the supervision process. 


X The program coordinator provides a regularly scheduled supervision time (individually) for specialists to 
consider how to perform their roles more effectively and how to support students and strengthen their 
networks more comprehensively. This differs from time spent doing administrative tasks. 


X The program coordinator shifts fluidly between the three roles (teacher, counselor, and consultant) in order 
to meet the specialist’s needs. 


X The program coordinator balances the focal topics (advocacy, mentoring, social-emotional support, and 
academic support), and does not overly emphasize one topic. 


X The program coordinator is attentive to changes in specialists’ behaviors with and reactions to [students], 
intrusive thoughts, and other signs of burnout and stress. 


X Specialists report that the program coordinator conveys empathy and understanding for specialists, 
students, and their network. 


X Specialists report that the supervision improves their practice and delivery of the program. 


Comments: 
- Individual supervision occurs approximately every other week. 


- Individual supervision helps the Specialist better support students by problem solving, sharing 
feedback and processing different scenarios. 


- Coordinator makes herself available for emergent questions or problem solving needs outside of 
supervision times. Coordinator is clear about availability during off work hours and how to contact her 
in the event of a Specialist or Student need. 


- Coordinator balances roles of teacher, counselor and consultant and addresses Specialists with 
empathy and understanding. 
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Ill. THE ROLE OF THE SPECIALIST: Specialists have a responsibility to students as advocates, mentors, 
and providers of social-emotional and academic support. 


The program coordinator can evaluate this entire section for each specialist and use this as an 
opportunity to provide feedback to specialists on role fulfillment. 


10. Advocacy — Evidence that specialists are effective advocates for students at this site. 


X Specialists can describe all of the five steps to advocacy. 


X Specialists can provide examples of each of the following: (1) the specialist’s advocacy for the student, (2) 
teaching the student to self-advocate, and (3) engaging in activities that build the capacity of the student’s 
network to advocate on the student’s behalf. 


Fostering Opportunities team engagement in professional development that ensures leadership and staff 
are highly knowledgeable about federal and local legislation, rules, and procedures governing the BID 
process. 


Comments: 
- Specialists can describe all five steps to advocacy. 


- Specialists recognize their approach to advocacy varies between students. Advocacy ultimately falls 
on a spectrum, ranging from the Specialist role-modeling advocacy for students; to developing self- 
advocacy skills within the student; and building capacity among the students network to advocate for 
the student. 


- Specialists encourage students to develop the skills to eventually advocate for themselves. 


- Specialists participate in BID meetings for students on their caseloads and provide education around 
the process as needed. 


- Coordinator develops and delivers all professional development related to the BID process to 
educators and human services partners. 


- The team participates in many professional development opportunities. 


- The team is currently building a child welfare website for educators to access as a toolkit. This 
website will include detailed information about the BID process. 


- When questions arise around local or federal legislation, policies or procedures, the team seeks 
guidance from appropriate leadership. 


11. Mentoring* — Evidence that specialists are effective mentors for students at this site. 


X Specialists document meetings with students at least three times per month on average. 
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X  Check-ins are mostly occurring in person with the occasional check-in over the phone or 
via video chat when the student is not able to meet with the specialist in person. 


X_ There is evidence that students are forming enduring emotional attachments with 
specialists as evidenced by continued engagement in the program (e.g., consistent 
attendance at check-ins; self-report of positive relationship with specialist). 


Comments: 
- The program expectation is that Specialists see students 1x every week. 


- As of Feb 25 2020, Specialists were checking in with students an average of 3.4 times per month. 


- There are many factors that contribute to missed meetings, including illness, school breaks or 
closures, placement changes, hospitalizations, etc. 


- The majority of meetings are happening in the school environment. Specialists also utilize home or 
community visits when appropriate. Recent CDC recommendations and district closures in response to Covid- 
19 require phone or video meetings with students. 


- Maintaining a consistent check in time each week is important to our students and they come to expect 
the meetings. 


- Specialists develop strong, positive relationships with students through consistent meetings and support. 
The rate at which these relationships develop varies by student. The program Happy Story Bank serves as 
evidence that students and Specialists form enduring emotional attachments. 


- The vast majority of students who are selected for the program choose to participate. In 1.5 years, only 5 
students have chosen not to engage in services. 


12. Social-Emotional Support* — Evidence that specialists are effective providers of social- 
emotional support at this site. 


21 Specialists consistently demonstrate advanced motivational interviewing techniques, such 
as developing discrepancies, rolling with resistance, and supporting self-advocacy through the 
use of affirmations targeted to the student’s locus of control. 


X Specialists consistently use basic motivational interviewing skills, such as open-ended 
questions and affirmations to build rapport with students. 


X Specialists primarily use strengths-based language to communicate with students. 
X Specialists use mostly person-first language, which is language that is neither stigmatizing 
nor objectifying (e.g., “students experiencing foster care” or “students in foster care” instead 


of “foster care students”) 


X Specialist appear to take a trauma-informed approach in all communications with students. 
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X Specialists can provide at least two examples of advocating for the trauma-informed approach (e.g., 
ensuring that the school environment does not unduly contribute to the student’s stress by guaranteeing that it 
is safe, supportive, and does not perpetuate trauma that the student has previously experienced). 


X Specialists never overstep boundaries into the role of mental health provider. 


Comments: 

- Specialists incorporate moderate to advanced level Motivational Interviewing skills into their 
conversations with students. Specialists recognizes MI as a skillset that develops with ongoing training 
and practice. 


- Specialists always use strengths-based and person-first language with students and all members of 
the students’ network. 


- Specialists use a trauma-informed lens when approaching any situation regarding their students, 
including their immediate environment. 


- Specialists often advocate for a trauma-informed approach from other educators and administrators 
when it comes to our students attendance, behavior or academic needs in school. 


- Specialists understand their roles and do not overstep into the role of mental health provider. 
- Specialists are very familiar with how to engage families in the conversation about seeking mental 


health treatment, how to complete a referral to a local agency, and how to approach mental health 
without stigma or shame. 


13. Academic Support — Evidence that specialists are effective providers of academic support at this site. 


X Specialists can provide examples of each of the following types of academic support: (1) academic support 
that creates bridges across systems; (2) academic support that addresses equity issues; (3) academic support 
that creates continuity through transitions; and (4) academic support that mentors students towards 
independence and self-advocacy. 


X On average, progress monitoring suggests that students are making progress toward their educational and 
career goals. 


X Academic support provided by specialists is responsive to the unique needs of each individual student but 
does not replace or supplant school-based supports or targeted interventions the student might be eligible to 
receive. 


Comments: 
- Specialists can identify examples of all types of academic support. 


- Specialists discuss students' academic progress at almost every meeting. 
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- Specialists ensure equitable access to learning for all students by identifying resources, making 
referrals and communicating with the students’ greater network. 


- Specialists work with students to identify barriers and collaborate with other school staff to problem 
solve ways to ensure students are supported in the best way possible. 


- Specialists are able to recognize when students are making progress towards their educational and 
career goals. 


- The progress monitoring reports are not a comprehensive reflection of the complete student 
experience. Often the progress is qualitative data that cannot be measured by the current 
progressmonitoring practices. When Specialists have important qualitative reflections to share, 
they utilize email or the progress monitoring narratives to share with the students’ network. 


- Specialists do not replace any other school staff or support. Specialists serve as additional support 
and a key member of the students’ network. 


*This indicator would be best assessed via observation of specialists’ check-ins with students, either in- 
person or via a secure remote method. 
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Fostering Opportunities Self-Assessment Checklist 


This is a self-assessment checklist that can be used by Fostering Opportunities leadership and staff and 
involved child welfare and education leadership to identify areas of strength and areas for improvement 
in the implementation of the Fostering Opportunities program. 


Rater: 2020/21 Fostering Opportunities Team 
Date: March 2021 
Program Location/Site: Jeffco Schools, Student Engagement Office 


I. SYSTEMS ALIGNMENT: Systems alignment refers to the education and child welfare policy and 
practice coordination that is the foundation for successful implementation of the Fostering 
Opportunities intervention. 


1. Leadership Framework — Evidence of a site’s leadership-driven culture of commitment to the Fostering 
Opportunities program and its principles. 


X Leaders within the child welfare and education agencies position the program for sustainability (e.g., 
support efforts to secure funds, partner in removing barriers, and prioritize internal resources as needed). 


X Leaders within the child welfare and education agencies engage with each other to create solutions when 
implementation challenges arise (e.g., articulate concrete examples). 


X Leaders participate in regularly scheduled meetings to launch and sustain the program. 


|_| Leaders spearheading the implementation have a high level of decision-making authority to prioritize 
practice changes necessary to align systems (e.g., no approval for decisions is needed). 


X Leaders ensure child welfare caseworkers and school leaders (e.g., principals) are familiar with the Fostering 
Opportunities program and how to make referrals. 

Comments: 

- The program is directly supported in the School District by the Director of the Student Engagement Office who 
is a direct report to the Chief of Student Success. Sustainability planning has been in progress since the advent 
of this program. Likewise, the program is supported by a Program Supervisor in DHS who is a direct report to 
the Executive Leadership team at Jeffco DHS who are aware of and invested in this programming. There is 
consistent participation in operations and governance meetings to create solutions and plan for sustainability by 
leadership in both organizations. 


- We purposefully did not check the ‘high level of decision making’ box as the disclaimer that ‘no approval for 
decisions is needed’ does not realistically represent well the systems level changes that this program has 
initiated. Based on the School District governance structure at times the improvements initiated by this 
program have required Cabinet, Superintendent and Board of Education approvals. 


- Leaders have ensured through their own practice as well as in the work of their teams that child welfare 
caseworkers and school leaders are familiar with this program. One caveat is that part of this work has 
been to inform stakeholders that they cannot make referrals due to the nature of the study design. 
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2. Legal Framework — Evidence of a site establishing a strong legal framework for Fostering Opportunities 
so that the program can function as smoothly as possible. 


X_ The site has established memorandums of understanding (MOUs) that clarify the responsibilities of the 
child welfare and education agencies within the Fostering Opportunities program, including related to the Best 
Interest Determination (BID) process. 


X The site has established data sharing agreements (DSAs) that detail what information can be shared 
between agencies for the purposes of delivering and evaluating the program. 


X The MOUs and DSAs are defined by the target population and geographical boundaries for the intervention 
(e.g., youth in foster care age 11 to 16 who are enrolled in a given school district). 


X The site has developed releases of information (ROI), which are student-specific documents that indicate 
what information can be shared, with whom, and for how long. ROls are on file for all participating students. 


Comments: 

- This program has an established contract that defines the target population and boundaries. The program has 
MOus as well as DSAs to allow for operations related to information sharing between agencies, BIDs, and 
student, caregiver access to information. There is work that still needs to be done to recognize the needs in 
these areas now that programming is operational as opposed to in its conceptual infancy. 


- This program has developed unique Release of Information Sharing protocols attentive to ensuring youth are 
empowered to make information sharing decisions and incorporates the appropriate agency specific releases to 
eliminate any barriers. 


3. Practice Framework — Evidence that day-to-day practices and procedures affecting students in the school 
environment are implemented with fidelity at this site. 


X The site has in place standard and, ideally, automated processes for notifying the Fostering Opportunities 
staff when a placement change occurs. 


X Fostering Opportunities’ role in the BID process is implemented as it is defined in the MOU, which may 
include: providing timely and accurate information on students’ educational progress and needs; planning for 
transitions when a school change is necessary; and ensuring communication between human services agencies 
and schools so that seamless transition occurs. 


Comments: 
- Coordinator receives all Notification of Placements from Jefferson County Human Services. 


- The Coordinator will share any notifications with appropriate Specialists when a program student is involved. 


- Specialists play a crucial role in the BID process, including (but not limited to) information sharing, transition 
planning and transportation logistics. 
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Il. PROGRAM CHARACTERISTICS: The site-specific characteristics, procedures, and plan of action in place 
to implement the Fostering Opportunities program. 


4. Staffing — Evidence that the Fostering Opportunities program is adequately staffed at this site. 


X There are a sufficient number of staff dedicated to Fostering Opportunities implementation at this site, 
including: child welfare and education leadership, program coordinators, and specialists. This is measured in 
terms of the number of FTEs needed to support the program’s complexity and size at this site. 
o The recommended ratio of students to specialists is 20 students or fewer per specialist. The ideal ratio 
of specialists to program coordinators is 10 specialists or fewer per program coordinator. 


X_ There is a strong plan in place to handle planned and unplanned specialist staffing transitions. 


Comments: 
- This site employs 1 Program Coordinator and 4 Specialists. 


- Coordinator is supervised by Student Engagement Office leadership, including the Director and Manager. 
- An Operations committee provides monthly oversight and includes leadership from local and state human 
service agencies, the school district, program evaluators and funding partners. - A Governance Committee 


meets quarterly review program benchmarks, financials and future planning. 


- Specialists have the capacity to caseload up to 20 students. Today, caseloads are approximately 75% full. 


5. Database — Evidence that a quality Fostering Opportunities database has been created and can be used 
to facilitate network closure at the site. 


X Adatabase specific to Fostering Opportunities has been created for the site with the required 
functionalities: 


Directory Functionality 


X Educational Progress Monitoring Functionality 
X Child Welfare Information Functionality 


Permissions Functionality 
Reporting Functionality 


Information in the database is up-to-date and accurate. 


The database adheres to all relevant federal and local data security requirements. 


X There is reasonable IT support provided for any technical issues that may arise. 
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Comments: 
- This site uses a CAT overlay designed specifically for Fostering Opportunities. 


- Specialists are able to access some of the functions listed above. Not all of the functions are available all of the 
time. 


- The Program Coordinator assumes, but cannot say for sure, that it adheres to all relevant federal and local 
data security requirements. 


- The Program Coordinator has access to reasonable IT support for any technical issues that may arise. The 
Program Coordinator might assume the modernization of TRAILS has taken most of the attention of the IT 
support team. 


6. Monthly Progress Monitoring Reports — Evidence that monthly progress monitoring reports are 
generated every month and shared with both the student and all members of the student’s network. 


X Aprogress monitoring report is generated by the database for all students every month. 


X Progress monitoring reports are shared and discussed with all students during at least one check-in per 
month. 


X Progress monitoring reports with students’ entire networks as listed on the social capital assessment (and 
the program coordinator may be copied on emails to the student’s network to ensure this is completed). 


X Progress monitoring reports use primarily strengths-based and trauma-informed language. 


Comments: 
A progress monitoring report is created by the Specialist each month (not generated by the database). The 
practice of using the database shifted when the learning environment transitioned to remote and the Fostering 
Opportunities team revamped the report to be more inclusive of this change. Specialists review the information 
from these reports with the student monthly. Specialists share the reports with the student’s network monthly, 
and upload the reports to the designated program file. The conversations with students about these reports, 
and the language used in the text of the reports is strengths based and trauma-informed. Program coordinator 
can view the reports from the file to ensure they are complete and access the information if needed. 

7. Network Closure — Evidence that the program is fulfilling its goal of facilitating network closure for 


participating students. 
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X There are consistent documented improvements in students’ social capital assessments over time (e.g., at 
the six-month point as compared to the one-month point there are more people listed in the “supports” column 
or the quality of the support has improved). 


X Monthly progress monitoring reports are sent to all members of students’ networks (and the program 
coordinator may be copied on emails to the student’s network to ensure this is completed). 


X Specialists demonstrate familiarity with all members of students’ network (e.g. names, details about 
relationship, etc.). 


X Specialists are consistently checking-in with multiple members of students’ networks at least once a month. 


Comments: 

- Specialists are familiar with and communicate with appropriate members of students’ network on an ongoing 
and consistent basis throughout the month. The Specialist employs a variety of ways to communicate to ensure 
information is accessible to everyone. 


- As students identify or are assigned additional members to their network, the Specialist will take necessary steps 
to get a signed ROI, obtain contact information, outreach to and include them in future communications. 


- Specialists always share completed progress monitoring reports at the end of each month with appropriate 
members of students’ network. 


- Completing a formal social capital assessment is not a current practice. Improvements in students’ social capital 
are recognized by the Specialist during meetings with the students and documented in case notes and progress 
monitoring reports. 


8. Equitable Access to the Program — Evidence that students are selected to participate in the program in 
an equitable and unbiased manner at this site. 


There is a start list from which to select students to the program that is comprised of those students in the 
school district who are also identified as having experienced foster care. If there are insufficient resources 
to serve all students in the district, participants are randomly selected into the program. 


OR 


Multiple methods are used to identify and recruit students so that the students who are selected to the 
program are not just those who have the strongest advocates in their networks. 
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Comments: 
- Neither of the above options accurately describe access to the program. 


- Students are selected to participate in an equitable and unbiased method that supports the evaluative nature 
of the current grant. 


- Students in grades 6th through 11th with open Dependency and Neglect cases with a local Human Services 
agency are filtered into the database. The Program Coordinator confirms enrollment in Jeffco Public Schools and 
then submits de-identified student information to the Colorado Evaluation & Action Lab via a secure link. 


- Students are assigned to the intervention or control group through a formal randomization process by the 
Colorado Evaluation & Action Lab. - Students identified for the intervention group are assigned to a Specialists 
caseload and services begin upon receipt of the appropriate county’s consent form. 


9. Supervision* — Specialists are receiving adequate supervision from the program coordinator, which 
enables them to better serve students. 


Note: The program coordinator can give a copy of this section to each specialist to fill out anonymously 
as a way to seek feedback on the supervision process. 


X The program coordinator provides a regularly scheduled supervision time (individually) for specialists to 
consider how to perform their roles more effectively and how to support students and strengthen their 
networks more comprehensively. This differs from time spent doing administrative tasks. 
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X The program coordinator shifts fluidly between the three roles (teacher, counselor, and consultant) in order 
to meet the specialist’s needs. 


X The program coordinator balances the focal topics (advocacy, mentoring, social-emotional support, and 
academic support), and does not overly emphasize one topic. 


X The program coordinator is attentive to changes in specialists’ behaviors with and reactions to [students], 
intrusive thoughts, and other signs of burnout and stress. 


X Specialists report that the program coordinator conveys empathy and understanding for specialists, 
students, and their network. 


X Specialists report that the supervision improves their practice and delivery of the program. 


Comments: 

The program coordinator provides comprehensive and consistent support to Specialists to enable them to best 
support students. The program coordinator schedules multiple supervision checks in with Specialists per month, 
as well as is available for problem-solving and thought-partnering with Specialists outside of scheduled 
supervision time. The program coordinator meets the defined roles of teacher, counselor, and consultant for 
the Specialists. The coordinator consistently discusses aspects of the focal topics of the program with 
Specialists. The program coordinator is aware of the Specialists social-emotional well-being, and recognizes the 
impacts of the role on stress and well-being. The program coordinator is empathetic, and supports Specialists, 
students, and their networks consistently both during supervision times, as well as team meetings, and in 
unscheduled conversations and interactions. The program coordinator’s support and guidance improves my 
practice and delivery of the program. 


Ill. THE ROLE OF THE SPECIALIST: Specialists have a responsibility to students as advocates, mentors, 
and providers of social-emotional and academic support. 


The program coordinator can evaluate this entire section for each specialist and use this as an 
opportunity to provide feedback to specialists on role fulfillment. 


10. Advocacy — Evidence that specialists are effective advocates for students at this site. 


X Specialists can describe all of the five steps to advocacy. 


X Specialists can provide examples of each of the following: (1) the specialist’s advocacy for the student, (2) 
teaching the student to self-advocate, and (3) engaging in activities that build the capacity of the student’s 
network to advocate on the student’s behalf. 


Fostering Opportunities team engagement in professional development that ensures leadership and staff 
are highly knowledgeable about federal and local legislation, rules, and procedures governing the BID 
process. 


Comments: 

Specialists can describe all five steps to advocacy and provide examples of advocating for a student (modeling 
and supporting through situations), supporting the student in self-advocacy in both the education and child 
welfare setting. During these remote times during the pandemic, Specialists have supported students in 
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advocating for themselves with members of their network by phone, video platform, and email to improve their 
self-advocacy. Specialists also provide examples and support to members of the students network to increase 
their advocacy for the student. Specialists inform and discuss federal and local legislation, rules, and procedures 
for students experiencing out-of-home care, as well as more specifically the BID process as needed when these 
topics come up in relation to the student, and Specialists ask for clarification from leadership if they are not 
familiar with specific legislation, rules and procedures, as an ongoing learning and professional development 
process. The team worked together with the coordinator to create a child welfare website which includes 
information on the BID process. Specialists have participated in BID meetings remotely, and ensured the BID 
meetings occur when there may be confusion or misunderstanding about if a BID meeting should occur during 
remote learning/hybrid learning times. 


11. Mentoring* — Evidence that specialists are effective mentors for students at this site. 


X Specialists document meetings with students at least three times per month on average. 


X  Check-ins are mostly occurring in person with the occasional check-in over the phone or via video chat 
when the student is not able to meet with the specialist in person. 


X_ There is evidence that students are forming enduring emotional attachments with specialists as evidenced 
by continued engagement in the program (e.g., consistent attendance at check-ins; self-report of positive 
relationship with specialist). 


Comments: 

Specialists meet with students either by phone, video platform, or in person for check ins at least 3 times per 
month with the intention of checking in weekly. Specialists meet much more frequently with students by phone 
or video during the pandemic, but are able to maintain consistent communication and rapport with students 
and provide support while remote. Specialists do still meet with students in-person occasionally when 
appropriate and aligned with current district and public health guidelines. Specialists continue to have 
experiences with students that demonstrate ongoing relationship and rapport, and most students have 
remained engaged in the program despite the challenges of the primarily remote environment. Some students 
also reach out to Specialists on their own outside of scheduled check-ins for support or to process both 
successes and challenges. 


12. Social-Emotional Support* — Evidence that specialists are effective providers of social-emotional support 
at this site. 


X Specialists consistently demonstrate advanced motivational interviewing techniques, such as developing 
discrepancies, rolling with resistance, and supporting self-advocacy through the use of affirmations targeted to 
the student’s locus of control. 


X Specialists consistently use basic motivational interviewing skills, such as open-ended questions and 
affirmations to build rapport with students. 


X Specialists primarily use strengths-based language to communicate with students. 


X Specialists use mostly person-first language, which is language that is neither stigmatizing nor objectifying 
(e.g., “students experiencing foster care” or “students in foster care” instead of “foster care students”) 


X Specialist appear to take a trauma-informed approach in all communications with students. 
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X Specialists can provide at least two examples of advocating for the trauma-informed approach (e.g., 
ensuring that the school environment does not unduly contribute to the student’s stress by guaranteeing that it 
is safe, supportive, and does not perpetuate trauma that the student has previously experienced). 


X Specialists never overstep boundaries into the role of mental health provider. 


Comments: 

Specialists have continued to participate in professional learning opportunities regarding motivational 
interviewing and use a variety of the techniques to support students based on the individual student situation 
and needs, and will continue to develop skills in these areas over time through continued professional 
development. Specialists consistently use strengths-based, asset-based, and person-first language to 
communicate with students, and to communicate about students in front of other professionals. Specialists 
approach their support of students through a trauma-informed lens, as well as bring this lens to advocacy for 
students, and can provide many examples of advocating for student from this lens, so other professionals 
understand and approach situations from the trauma-informed lens. Specialists understand our role as different 
from that of a mental health provider, and refer students and families to mental health support when 
appropriate, as well as approach discussions of mental health related topics in a way that provides support to 
the student’s social-emotional well-being without providing mental health services. 


13. Academic Support — Evidence that specialists are effective providers of academic support at this site. 


X Specialists can provide examples of each of the following types of academic support: (1) academic support 
that creates bridges across systems; (2) academic support that addresses equity issues; (3) academic support 
that creates continuity through transitions; and (4) academic support that mentors students towards 
independence and self-advocacy. 


X On average, progress monitoring suggests that students are making progress toward their educational and 
career goals. 


X Academic support provided by specialists is responsive to the unique needs of each individual student but 
does not replace or supplant school-based supports or targeted interventions the student might be eligible to 
receive. 


Comments: 

Specialists can provide examples of all of the types of academic support in different situations with different 
students. Specialists discuss academic progress and goals with students and their teams ongoing, and students 
frequently demonstrate progress towards their individual goals, even when not demonstrated in quantitative 
evidence in their grades, but in qualitative evidence of their progress and growth as individuals. Specialists often 
advocate for the student to receive additional supports and interventions from other school-based staff, such as 
when we advocate for 504 or IEP evaluations and reviews, advocate with individual teachers regarding a 
student’s needs for a specific class, advocate for tutoring if a student’s needs are outside of the responsibilities 
and the school-based team. The Specialists and Coordinator have also informed some district-wide suggestions 
around engaging students in remote learning, and accommodations that can be used by school-based teams to 
support students participating in the program, as well as all students based on individual situations and needs. 


*This indicator would be best assessed via observation of specialists’ check-ins with students, either in- 
person or via a secure remote method. 
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